FEAT: Add arrow fetch support by ffelixg · Pull Request #354 · microsoft/mssql-python

ffelixg · 2025-11-30T21:00:35Z

Work Item / Issue Reference

GitHub Issue: #130

Summary

Hey, you mentioned in issue #130 that you were willing to consider community contributions for adding Apache Arrow support, so here you go. I have focused only on fetching data into Arrow structures from the Database.

The Function signatures I chose are:

arrow_batch(chunk_size=10000): Fetch a single pyarrow.RecordBatch, base for the other two methods.
arrow(chunk_size=10000): Fetches the entire result set as a single pyarrow.Table.
arrow_reader(chunk_size=10000): Returns a pyarrow.RecordBatchReader for streaming results without loading the entire dataset into RAM.

Using fetch_arrow... instead of just arrow... could also be a good option, but I think the terse version is not too ambiguous.

Technical details

I am not very familiar with C++, but I did have some prior practice for this task from implementing my own ODBC driver in Zig (a very good language for projects like this!). The implementation is written almost entirely in C++ in the FetchArrowBatch_wrap function, which produces PyCapsules that are then consumed by arrow_batch and turned into actual arrow objects.

The function itself is very large. I'm sure it could be factored in a better way, even sharing some code with the other methods of fetching, but my goal was to keep the whole thing as straight forward as possible.

I have also implemented my own loop for SQLGetData for Lob-Columns. Unlike with the python fetch methods, I don't use the result directly, but instead copy it into the same buffer I would use for the case with bound columns. Maybe that's an abstraction that would make sense for that case as well.

Notes on data types

I noticed that you use SQL_C_TYPE_TIME for time(x) columns. The arrow fetch does the same, but I think it would be better to use SQL_C_SS_TIME2, since that supports fractional seconds.

Datetimeoffset is a bit tricky, since SQL Server stores timezone information alongside each cell, while arrow tables expect a fixed timezone for the entire column. I don't really see any solution other than converting everything to UTC and returning a UTC column, so that's what I did.

SQL_C_CHAR columns get copied directly into arrow utf8 arrays. Maybe some encoding options would be useful.

Performance

I think the main performance win to be gained is not interacting with any Python data structures in the hot path. That is satisfied. Further optimizations, which I did not make are:

Releasing the GIL for the entire fetch loop
Sharing the bound fetch buffer across repeated fetch calls
Improve the hot loop switching

Instead of looping over rows and columns and then switching on the data type for each cell, you could

Put the row loop inside each switch case (fastest I think, but would bloat the code a lot more)
Use function pointers like you recently did for python fetching (has overhead because of the indirect function call I think, also code is more scattered)
Replace both loops and the switch with computed gotos. That's what I opted for in my ODBC driver (the Zig equivalent is a labeled switch) and I am quite happy with how it came out. Performance seems very good and it allows you to abstract the fetching process on a row by row basis. I don't know how well that would translate to C++.

Overall the arrow performance seems not too far off from what I achieved with zodbc.

ffelixg · 2025-11-30T21:03:39Z

@microsoft-github-policy-service agree

Copilot

Pull request overview

This PR adds Apache Arrow fetch support to the mssql-python driver, enabling efficient columnar data retrieval from SQL Server. The implementation provides three new cursor methods (arrow_batch(), arrow(), and arrow_reader()) that convert result sets into Apache Arrow data structures using the Arrow C Data Interface, bypassing Python object creation in the hot path for improved performance.

Key changes:

Implemented Arrow fetch functionality in C++ that directly converts ODBC result sets to Arrow format
Added three Python API methods for different Arrow data consumption patterns (single batch, full table, streaming reader)
Added comprehensive test coverage for various data types, LOB columns, and edge cases

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 9 comments.

File	Description
mssql_python/pybind/ddbc_bindings.cpp	Core C++ implementation: Added `FetchArrowBatch_wrap()` function with Arrow C Data Interface structures, column buffer management, data type conversion logic, and memory management for Arrow structures
mssql_python/cursor.py	Python API layer: Added `arrow_batch()`, `arrow()`, and `arrow_reader()` methods that wrap the C++ bindings and handle pyarrow imports
tests/test_004_cursor.py	Comprehensive test suite covering wide tables, LOB columns, individual data types, empty result sets, datetime handling, and batch operations
requirements.txt	Added pyarrow as a dependency for development and testing

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mssql_python/pybind/ddbc_bindings.cpp

mssql_python/cursor.py

mssql_python/pybind/ddbc_bindings.cpp

sumitmsft · 2025-12-01T11:14:32Z

Work Item / Issue Reference

GitHub Issue: #130

Summary

Hey, you mentioned in issue #130 that you were willing to consider community contributions for adding Apache Arrow support, so here you go. I have focused only on fetching data into Arrow structures from the Database.

The Function signatures I chose are:

arrow_batch(chunk_size=10000): Fetch a single pyarrow.RecordBatch, base for the other two methods.

arrow(chunk_size=10000): Fetches the entire result set as a single pyarrow.Table.

arrow_reader(chunk_size=10000): Returns a pyarrow.RecordBatchReader for streaming results without loading the entire dataset into RAM.

Using fetch_arrow... instead of just arrow... could also be a good option, but I think the terse version is not too ambiguous.

Technical details

I am not very familiar with C++, but I did have some prior practice for this task from implementing my own ODBC driver in Zig (a very good language for projects like this!). The implementation is written almost entirely in C++ in the FetchArrowBatch_wrap function, which produces PyCapsules that are then consumed by arrow_batch and turned into actual arrow objects.

The function itself is very large. I'm sure it could be factored in a better way, even sharing some code with the other methods of fetching, but my goal was to keep the whole thing as straight forward as possible.

I have also implemented my own loop for SQLGetData for Lob-Columns. Unlike with the python fetch methods, I don't use the result directly, but instead copy it into the same buffer I would use for the case with bound columns. Maybe that's an abstraction that would make sense for that case as well.

Notes on data types

I noticed that you use SQL_C_TYPE_TIME for time(x) columns. The arrow fetch does the same, but I think it would be better to use SQL_C_SS_TIME2, since that supports fractional seconds.

Datetimeoffset is a bit tricky, since SQL Server stores timezone information alongside each cell, while arrow tables expect a fixed timezone for the entire column. I don't really see any solution other than converting everything to UTC and returning a UTC column, so that's what I did.

SQL_C_CHAR columns get copied directly into arrow utf8 arrays. Maybe some encoding options would be useful.

Performance

I think the main performance win to be gained is not interacting with any Python data structures in the hot path. That is satisfied. Further optimizations, which I did not make are:

Releasing the GIL for the entire fetch loop

Sharing the bound fetch buffer across repeated fetch calls

Improve the hot loop switching

Instead of looping over rows and columns and then switching on the data type for each cell, you could

Put the row loop inside each switch case (fastest I think, but would bloat the code a lot more)

Use function pointers like you recently did for python fetching (has overhead because of the indirect function call I think, also code is more scattered)

Replace both loops and the switch with computed gotos. That's what I opted for in my ODBC driver (the Zig equivalent is a labeled switch) and I am quite happy with how it came out. Performance seems very good and it allows you to abstract the fetching process on a row by row basis. I don't know how well that would translate to C++.

Overall the arrow performance seems not too far off from what I achieved with zodbc.

Hi @ffelixg

Thanks for raising this PR. Please allow us time to review and share our comments.

Appreciate your diligence in strengthening this project.

Sumit

mssql_python/pybind/ddbc_bindings.cpp

sumitmsft · 2025-12-04T11:32:31Z

Hello @ffelixg

Me and my team are in the process of reviewing your PR. While we are getting started, it would be great to have some preliminary information from you on the following items:

Have you created any design document for this feature (high\low level)? Could you please attach it here or share it with us at the below mentioned email id?
What is your motivation to bring the support for Arrow in mssql-python? Could you help us understand the use case(s) you're trying to address?
Is there a way to connect with you over Microsoft Teams call, so that we can closely work on this feature together? You can reach out to us at mssql-python@microsoft.com with your contact details and consent to connect with you.

Regards,
Sumit

ffelixg · 2025-12-04T20:31:24Z

Hello @sumitmsft,

I'm happy to hear that.

I don't have any design document beyond what I wrote in the PR description. Are there any areas in particular you would like me to provide more information on?
I assume the motivation is mostly in line with what most arrow users like about arrow. Mainly I believe that arrow is the correct format for anything that is using batches of data and has C-Extensions for both producer and consumer. For example arrow gives you great interop with things like duckdb, polars, pandas on the analytics/ML side. Also I want python to be the obvious one stop shop for ETL workloads and for that, plain python types don't work well both for performance and reliability. You still have plenty of situations though where you want to fetch one result set with python types and the next with arrow types, so it has to be in the same driver as well.
Yes, for sure. I have sent you an Email.

Regards,
Felix

bewithgaurav · 2025-12-08T11:53:57Z

/azp run

mssql_python/pybind/ddbc_bindings.cpp

azure-pipelines · 2025-12-08T11:54:09Z

Azure Pipelines successfully started running 1 pipeline(s).

bewithgaurav

@ffelixg - Thanks for the contribution! :)
Before we get started on this PR - there are a few dev build workflows we need to fix.
Could you please take a look at the Azure DevOps Workflows which are failing? (goes by the check MSSQL-Python-PR-Validation):

Build issues on windows (ref)
A few tests failing on MacOS (ref)

ffelixg · 2025-12-09T21:52:05Z

Hey,

the Windows issue was due to me using a 128 bit integer type which isn't supported by MSVC. To address that, I've added a custom 128 bit int type implemented as two 64 bit ints with some overloaded operators. I'm not super happy about that, it seems to me that using an existing library for this type of thing would be better. If you prefer to use a library, I'd leave the choice of which one to use up to you though.

The 128 bit type is only needed for decimals, so an alternative solution would be to use the numeric struct instead of strings for fetching. That one has near identical bit-representation compared to arrow and wouldn't require a lot of modification. But that's a change that would affect fetching to Python objects as well, since the two paths should probably stay in sync.

The MacOS issue seems to be due to std::mktime failing. I've added an implementation of days_from_civil to eliminate that call. I think a newer version of c++ would include that function. CPython also has an implementation for that in _datetimemodule.c, but it sadly isn't directly accessible, only when going through python objects, which would of course be slow.

I noticed some CI errors related to zip not taking strict=True, which only applies for Python 3.9 and below. I know you don't officially support 3.9, but if you're testing it and it works otherwise, I could write that differently I guess.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…by private_data

…ring ownership to arrow

gargsaumya · 2026-01-06T07:05:24Z

/azp run

azure-pipelines · 2026-01-06T07:05:34Z

Azure Pipelines successfully started running 1 pipeline(s).

github-actions · 2026-01-06T07:24:59Z

📊 Code Coverage Report

🔥 Diff Coverage 85%	🎯 Overall Coverage 77%	📈 Total Lines Covered: `6246` out of `8063` 📁 Project: `mssql-python`

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

mssql_python/cursor.py (80.0%): Missing lines 39,2416,2420-2421,2444-2445,2471-2472
mssql_python/pybind/ddbc_bindings.cpp (85.4%): Missing lines 4234-4235,4262-4263,4270-4271,4280,4294-4295,4330-4332,4375,4422,4442,4459-4460,4472-4478,4515-4517,4537-4539,4564-4566,4580-4582,4586,4597-4599,4611-4613,4625-4627,4639-4641,4653-4655,4659-4671,4682-4684,4697-4699,4713-4715,4727-4729,4741-4743,4747,4757-4759,4771-4773,4785-4787,4791-4796,4813,4834-4836,4847-4848,4854,4861-4862,4868-4869,4904-4905,4937,4960,4999-5000,5010,5028-5036,5120-5121,5125-5127,5218-5219,5223-5225

Summary

Total: 950 lines
Missing: 141 lines
Coverage: 85%

mssql_python/cursor.py

Lines 35-43

Lines 2412-2425

Lines 2440-2449

Lines 2467-2476

mssql_python/pybind/ddbc_bindings.cpp

Lines 4230-4239

0 ""